Mean System Usability Scale Score Research Articles

User-Centered Development of a Patient Decision Aid for Choice of Early Abortion Method: Multi-Cycle Mixed Methods Study.

People seeking abortion in early pregnancy have the choice between medication and procedural options for care. The choice is preference-sensitive-there is no clinically superior option and the choice depends on what matters most to the individual patient. Patient decision aids (PtDAs) are shared decision-making tools that support people in making informed, values-aligned health care choices. We aimed to develop and evaluate the usability of a web-based PtDA for the Canadian context, where abortion care is publicly funded and available without legal restriction. We used a systematic, user-centered design approach guided by principles of integrated knowledge translation. We first developed a prototype using available evidence for abortion seekers' decisional needs and the risks, benefits, and consequences of each option. We then refined the prototype through think-aloud interviews with participants at risk of unintended pregnancy ("patient" participants). Interviews were audio-recorded and documented through field notes. Finally, we conducted a web-based survey of patients and health care professionals involved with abortion care, which included the System Usability Scale. We used content analysis to identify usability issues described in the field notes and open-ended survey questions, and descriptive statistics to summarize participant characteristics and close-ended survey responses. A total of 61 individuals participated in this study. Further, 11 patients participated in think-aloud interviews. Overall, the response to the PtDA was positive; however, the content analysis identified issues related to the design, language, and information about the process and experience of obtaining abortion care. In response, we adapted the PtDA into an interactive website and revised it to include consistent and plain language, additional information (eg, pain experience narratives), and links to additional resources on how to find an abortion health care professional. In total, 25 patients and 25 health care professionals completed the survey. The mean System Usability Scale score met the threshold for good usability among both patient and health care professional participants. Most participants felt that the PtDA was user-friendly (patients: n=25, 100%; health care professionals: n=22, 88%), was not missing information (patients: n=21, 84%; health care professionals: n=18, 72%), and that it was appropriate for patients to complete the PtDA before a consultation (patients: n=23, 92%; health care professionals: n=23, 92%). Open-ended responses focused on improving usability by reducing the length of the PtDA and making the website more mobile-friendly. We systematically designed the PtDA to address an unmet need to support informed, values-aligned decision-making about the method of abortion. The design process responded to a need identified by potential users and addressed unique sensitivities related to reproductive health decision-making.

Assessing Differences in mHealth Usability and App Experiences Among Young African American Women: Secondary Analysis of a Randomized Controlled Trial.

In North Carolina, HIV continues to disproportionately affect young African American women. Although mobile health (mHealth) technology appears to be a tool capable of making public health information more accessible for key populations, previous technology use and social determinants may impact users' mHealth experiences. The objective of this study was to evaluate mHealth usability, assessing differences based on previous technology use and social determinants among a sample of African American women in emerging adulthood. As part of a National Institute on Drug Abuse-funded randomized controlled trial with African American women (aged 18-25 years), counties were assigned to receive an evidence-based HIV risk reduction intervention through mHealth and participants were asked to complete usability surveys at 6- and 12-month follow-ups. Participants' first survey responses were analyzed through 2-tailed t tests and linear regression models to examine associations with previous technology use and social determinants (P<.05). The mean System Usability Scale (SUS) score was 69.2 (SD 17.9; n=159), which was higher than the threshold of acceptability (68.0). Participants who had previously used a tablet indicated higher usability compared to participants without previous use (mean 72.9, SD 18.1 vs mean 57.6, SD 11.4; P<.001), and participants with previous smartphone use also reported higher usability compared to participants without previous use (mean 71.9, SD 18.3 vs mean 58.0, SD 10.7; P<.001). Differences in SUS scores were observed among those reporting homelessness (mean 58.3, SD 19.0 vs mean 70.8, SD 17.2; P=.01), unemployment (mean 65.9, SD 17.2 vs mean 71.6, SD 18.1; P=.04), or current school enrollment (mean 73.2, SD 18.5 vs mean 65.4, SD 16.5; P=.006). Statistically significant associations were not observed for food insecurity (mean 67.3, SD 18.6 vs mean 69.9, SD 17.7; P=.45). Although above-average usability was observed overall, these findings demonstrate differences in mHealth usability based on past and current life experiences. As mHealth interventions become more prevalent, these findings may have important implications for ensuring that mHealth apps improve the reach of evidence-based interventions. ClinicalTrials.gov NCT02965014; https://clinicaltrials.gov/study/NCT02965014. RR2-10.1186/s12889-018-5796-8.

A Randomized Evaluation of MoodFX, a Patient-Centred e-Health Tool to Support Outcome Measurement for Depression: Une évaluation randomisée de MoodFX, un outil de santé en ligne centré sur le patient pour soutenir la mesure du résultat dans la dépression.

e-Health tools using validated questionnaires to assess outcomes may facilitate measurement-based care for psychiatric disorders. MoodFX was created as a free online symptom tracker to support patients for outcome measurement in their depression treatment. We conducted a pilot randomized evaluation to examine its usability, and clinical utility. Patients presenting with a major depressive episode (within a major depressive or bipolar disorder) were randomly assigned to receive either MoodFX or a health information website as the intervention and control condition, respectively, with follow-up assessment surveys conducted online at baseline, 8 weeks and 6 months. The primary usability outcomes included the percentage of patients with self-reported use of MoodFX 3 or more times during follow up (indicating minimally adequate usage) and usability measures based on the System Usability Scale (SUS). Secondary clinical outcomes included the Quick Inventory of Depressive Symptomatology, Self-Rated (QIDS-SR) and Patient Health Questionnaire (PHQ-9). Forty-nine participants were randomized (24 to MoodFX and 25 to the control condition). Of the 23 participants randomized to MoodFX who completed the user survey, 18 (78%) used MoodFX 3 or more times over the 6 months of the study. The mean SUS score of 72.7 (65th-69th percentile) represents good usability. Compared to the control group, the MoodFX group had significantly better improvement on QIDS-SR and PHQ-9 scores, with large effect sizes and higher response rates at 6 months. There were no differences between conditions on other secondary outcomes such as functioning and quality of life. MoodFX demonstrated good usability and was associated with reduction in depressive symptoms. This pilot study supports the use of digital tools in depression treatment.

Unsupervised robot-assisted rehabilitation after stroke: feasibility, effect on therapy dose, and user experience

BackgroundUnsupervised robot-assisted rehabilitation is a promising approach to increase the dose of therapy after stroke, which may help promote sensorimotor recovery without requiring significant additional resources and manpower. However, the unsupervised use of robotic technologies is not yet a standard, as rehabilitation robots often show low usability or are considered unsafe to be used by patients independently. In this paper we explore the feasibility of unsupervised therapy with an upper limb rehabilitation robot in a clinical setting, evaluate the effect on the overall therapy dose, and assess user experience during unsupervised use of the robot and its usability.MethodsSubacute stroke patients underwent a four-week protocol composed of daily 45 min-sessions of robot-assisted therapy. The first week consisted of supervised therapy, where a therapist explained how to interact with the device. The second week was minimally supervised, i.e., the therapist was present but intervened only if needed. After this phase, if participants learnt how to use the device, they proceeded to two weeks of fully unsupervised training. Feasibility, dose of robot-assisted therapy achieved during unsupervised use, user experience, and usability of the device were evaluated. Questionnaires to evaluate usability and user experience were performed after the minimally supervised week and at the end of the study, to evaluate the impact of therapists’ absence.ResultsUnsupervised robot-assisted therapy was found to be feasible, as 12 out of the 13 recruited participants could progress to unsupervised training. During the two weeks of unsupervised therapy participants on average performed an additional 360 min of robot-assisted rehabilitation. Participants were satisfied with the device usability (mean System Usability Scale scores > 79), and no adverse events or device deficiencies occurred.ConclusionsWe demonstrated that unsupervised robot-assisted therapy in a clinical setting with an actuated device for the upper limb was feasible and can lead to a meaningful increase in therapy dose. These results support the application of unsupervised robot-assisted therapy as a complement to usual care in clinical settings and pave the way to its application in home settings.Trial registrationRegistered on 13.05.2020 on clinicaltrials.gov (NCT04388891).

Evaluation of an mHealth App on Self-Management of Osteoporosis: Prospective Survey Study.

Mobile health (mHealth) technologies can be used for disease-specific self-management, and these technologies are experiencing rapid growth in the health care industry. They use mobile devices, specifically smartphone apps, to enhance and support medical and public health practices. In chronic disease management, the use of apps in the realm of mHealth holds the potential to improve health outcomes. This is also true for mHealth apps on osteoporosis, but the usage and patients' experiences with these apps are underexplored. This prospective survey study aimed to investigate the eHealth literacy of Danish patients with osteoporosis, as well as the usability and acceptability of the app "My Bones." Data on patient characteristics, disease knowledge, eHealth literacy, usability, and acceptability were collected using self-administered questionnaires at baseline, 2 months, and 6 months. The following validated questionnaires were used: eHealth Literacy Questionnaire, System Usability Scale, and Service User Technology Acceptability Questionnaire. Mean scores for eHealth literacy ranged from 2.6 to 3.1, with SD ranging from 0.5 to 0.6 across the 7 domains. The mean (SD) System Usability Scale score was 74.7 (14.4), and the mean (SD) scores for domains 1, 2, and 6 of the Service User Technology Acceptability Questionnaire were 3.4 (1.2), 4.5 (1.1), 4.1 (1.2), respectively. Danish patients with osteoporosis are both motivated and capable of using digital health services. The app's usability was acceptable, and it has the potential to reduce visits to general practitioner clinics, enhance health outcomes, and serve as a valuable addition to regular health or social care services.

Adapting mHealth Interventions (PrEPmate and DOT Diary) to Support PrEP Retention in Care and Adherence Among English and Spanish-Speaking Men Who Have Sex With Men and Transgender Women in the United States: Formative Work and Pilot Randomized Trial.

A growing number of mobile health (mHealth) technologies are being developed to support HIV preexposure prophylaxis (PrEP) adherence and persistence; however, most tools have focused on men who have sex with men (MSM), and few are available in Spanish. To maximize the potential impact of these tools in reducing gender and racial/ethnic disparities and promoting health equity, mHealth tools tailored to Spanish-speaking people and transgender women are critically needed. The aim of this study is to adapt and tailor 2 mHealth technologies, PrEPmate and DOT Diary, to support daily PrEP adherence and persistence among Spanish-speaking MSM and English- and Spanish-speaking transgender women and to evaluate the feasibility and acceptability of these tools. PrEPmate, an interactive, bidirectional, text messaging intervention that promotes personalized communication between PrEP users and providers, and DOT Diary, a mobile app that promotes self-management of PrEP use and sexual health through an integrated electronic pill-taking and sexual activity diary, were previously developed for English-speaking MSM. We conducted 3 focus groups with 15 English- and Spanish-speaking transgender women and MSM in San Francisco and Miami to culturally tailor these tools for these priority populations. We then conducted a 1-month technical pilot among 21 participants to assess the usability and acceptability of the adapted interventions and optimize the functionality of these tools. Participants in focus groups liked the "human touch" of text messages in PrEPmate and thought it would be helpful for scheduling appointments and asking questions. They liked the daily reminder messages, especially the fun facts, gender affirmations, and transgender history topics. Participants recommended changes to tailor the language and messages for Spanish-speaking and transgender populations. For DOT Diary, participants liked the adherence tracking and protection level feedback and thought the calendar functions were easy to use. Based on participant recommendations, we tailored language within the app for Spanish-speaking MSM and transgender women, simplified the sexual diary, and added motivational badges. In the technical pilot of the refined tools, mean System Usability Scale scores were 81.2/100 for PrEPmate and 76.4/100 for DOT Diary (P=.48), falling in the "good" to "excellent" range, and mean Client Satisfaction Questionnaire scores were 28.6 and 28.3 for PrEPmate and DOT Diary, respectively (maximum possible score=32). Use of both tools was high over the 1-month pilot (average of 10.5 messages received from each participant for PrEPmate; average of 17.6 times accessing the DOT Diary app), indicating good feasibility for both tools. Using a user-centered design approach, we culturally tailored PrEPmate and DOT Diary to support daily PrEP use among Spanish-speaking MSM and English- and Spanish-speaking transgender women. Our positive findings in a technical pilot support further testing of these mHealth interventions in an upcoming comparative effectiveness trial.

Patient-Centered Chronic Wound Care Mobile Apps: Systematic Identification, Analysis, and Assessment.

The prevalence of chronic wounds is predicted to increase within the aging populations in industrialized countries. Patients experience significant distress due to pain, wound secretions, and the resulting immobilization. As the number of wounds continues to rise, their adequate care becomes increasingly costly in terms of health care resources worldwide. eHealth support systems are being increasingly integrated into patient care. However, to date, no systematic analysis of such apps for chronic wounds has been published. The aims of this study were to systematically identify and subjectively assess publicly available German- or English-language mobile apps for patients with chronic wounds, with quality assessments performed by both patients and physicians. Two reviewers independently conducted a systematic search and assessment of German- or English-language mobile apps for patients with chronic wounds that were available in the Google Play Store and Apple App Store from April 2022 to May 2022. In total, 3 apps met the inclusion and exclusion criteria and were reviewed independently by 10 physicians using the German Mobile App Rating Scale (MARS) and the System Usability Scale (SUS). The app with the highest mean MARS score was subsequently reviewed by 11 patients with chronic wounds using the German user version of the MARS (uMARS) and the SUS. Additionally, Affinity for Technology Interaction (ATI) scale scores were collected from both patients and physicians. This study assessed mobile apps for patients with chronic wounds that were selected from a pool of 118 identified apps. Of the 73 apps available in both app stores, 10 were patient oriented. After excluding apps with advertisements or costs, 3 apps were evaluated by 10 physicians. Mean MARS scores ranged from 2.64 (SD 0.65) to 3.88 (SD 0.65) out of 5, and mean SUS scores ranged from 50.75 (SD 27) to 80.5 (SD 17.7) out of 100. WUND APP received the highest mean MARS score (mean 3.88, SD 0.65 out of 5) among physicians. Hence, it was subsequently assessed by 11 patients and achieved a similar rating (uMARS score: mean 3.89, SD 0.4 out of 5). Technical affinity, as measured with the ATI scale, was slightly lower in patients (score: mean 3.62, SD 1.35 out of 6) compared to physicians (score: mean 3.88, SD 1.03 out 6). The quality ratings from physicians and patients were comparable and indicated mediocre app quality. Technical affinity, as assessed by using the ATI scale, was slightly lower for patients. Adequate apps for patients with chronic wounds remain limited, emphasizing the need for improved app development to meet patient needs. The ATI scale proved valuable for assessing technical affinity among different user groups.

Design and Usability of an Open-Source, Low-Cost Flexible Laryngoscope for Resource-Limited Settings

Endoscopes are paramount to the practice of otolaryngology. To provide physicians in low-middle-income countries with adequate tools to treat otolaryngologic problems, it is necessary to create a low-cost sustainable option. To describe the design and usability of an open-source, low-cost flexible laryngoscope that addresses the lack of affordable and accessible methods for otolaryngologic visualization in resource-limited settings. This quality improvement study used a mixed-methods approach, including a technical description of device design as well as quantitative and qualitative survey evaluation of device usability. Engineering involved device design, sourcing or manufacturing individual components, fabricating a prototype, and iterative testing. Key assumptions and needs for the device were identified in collaboration with otolaryngologists in Zimbabwe, and designed and simulated by biomedical engineers in a US university laboratory. Board-certified otolaryngologists at a single US university hospital trialed a completed prototype on simulated airways between May 2023 and June 2023. Technical details on the design of the device are provided. Otolaryngologist gave feedback on device characteristics, maneuverability, and visualization using the System Usability Scale, a customized Likert-scale questionnaire (5-point scale), and semistructured interviews. A functional prototype meeting requirements was completed consisting of a distal-chip camera, spring bending tip, handle housing the control mechanism and electronics, and flexible polyether block amide-coated silicone sheath housing the camera and control wires; an external monitor provided real-time visualization and ability to store data. A total of 14 otolaryngologists participated in the device review. The mean (SD) System Usability Scale score was 88.93 (10.08), suggesting excellent usability. The device was rated highly for ease of set up, physical attributes, image quality, and functionality. This quality improvement study described the design of a novel open-source low-cost flexible laryngoscope that external review with otolaryngologists suggests was usable and feasible in various resource-limited environments. Future work is needed to translate the model into a clinical setting.

Vanderbilt Electronic Health Record Voice Assistant Supports Clinicians.

Electronic health records (EHRs) present navigation challenges due to time-consuming searches across segmented data. Voice assistants can improve clinical workflows by allowing natural language queries and contextually aware navigation of the EHR. To develop a voice-mediated EHR assistant and interview providers to inform its future refinement. The Vanderbilt EHR Voice Assistant (VEVA) was developed as a responsive web application and designed to accept voice inputs and execute the appropriate EHR commands. Fourteen providers from Vanderbilt Medical Center were recruited to participate in interactions with VEVA and to share their experience with the technology. The purpose was to evaluate VEVA's overall usability, gather qualitative feedback, and detail suggestions for enhancing its performance. VEVA's mean system usability scale score was 81 based on the 14 providers' evaluations, which was above the standard 50th percentile score of 68. For all five summaries evaluated (overview summary, A1C results, blood pressure, weight, and health maintenance), most providers offered a positive review of VEVA. Several providers suggested modifications to make the technology more useful in their practice, ranging from summarizing current medications to changing VEVA's speech rate. Eight of the providers (64%) reported they would be willing to use VEVA in its current form. Our EHR voice assistant technology was deemed usable by most providers. With further improvements, voice assistant tools such as VEVA have the potential to improve workflows and serve as a useful adjunct tool in health care.

Supportive Digital Health Service During Cancer Chemotherapy: Single-Arm Before-and-After Feasibility Study.

Digital supportive cancer care is recommended to improve patient outcomes. A portal was designed and embedded within the electronic medical record and public health portal of Iceland, consisting of symptom and needs monitoring, educational material, and messaging. This study aims to assess (1) portal feasibility (adoption, engagement, usability, and acceptability), (2) potential predictors of usability and acceptability, and (3) the potential impact of the portal on patient-reported outcomes. This was a single-arm, before-and-after feasibility study at a university hospital among patients with cancer who were undergoing chemotherapy. Participation included filling out the Edmonton Symptom Assessment System-Revised (ESASr) weekly and the Distress Thermometer and Problem List (DT&PL) 3 times; reading educational material and messaging; and completing study questionnaires. Clinical and portal engagement data were collected from medical records. Data from patients were collected electronically at baseline and 7 to 10 days after the third chemotherapy round. Usability was assessed using the System Usability Scale (score 0-100), and acceptability was assessed using a 35-item survey (score 1-5). Patient-reported outcome measures included ESASr and DT&PL; a single-item scale for quality of life, family support, and quality of care; and multi-item scales for health literacy (Brief Health Literacy Screener), health engagement (Patient Health Engagement Scale), self-care self-efficacy (Self-Care Self-Efficacy scale), symptom interference (MD Anderson Symptom Inventory), knowledge expectations (Hospital Patients' Knowledge Expectations), and received knowledge (Hospital Patients' Received Knowledge). Health care professionals were interviewed regarding portal feasibility. The portal adoption rate was 72% (103/143), and the portal use rate was 76.7% (79/103) over a mean 8.6 (SD 2.7) weeks. The study completion rate was 67% (69/103). The combined completion rate of the ESASr and DT&PL was 78.4% (685/874). Patients received a mean 41 (SD 13) information leaflets; 33% (26/79) initiated messaging, 73% (58/79) received messages, and 85% (67/79) received follow-up phone calls. The mean System Usability Scale score was 72.3 (SD 14.7), indicating good usability. Usability was predicted by age (β=-.45), ESASr engagement (β=.5), symptom interference (β=.4), and received knowledge (β=.41). The mean acceptability score, 3.97 (SD 0.5), was above average and predicted by age (β=-.31), ESASr engagement (β=.37), symptom interference (β=.60), self-care self-efficacy (β=.37), and received knowledge (β=.41). ESASr scores improved for total symptom distress (P=.003; Cohen d=0.36), physical symptoms (P=.01; Cohen d=0.31), and emotional symptoms (P=.01; Cohen d=0.31). Daily symptom interference increased (P=.03; Cohen d=0.28), quality of life improved (P=.03; Cohen d=0.27) and health engagement (P=.006; Cohen d=0.35) improved, while knowledge expectations decreased (P≤.001; Cohen d=2.57). Health care professionals were positive toward the portal but called for clearer role delineation and follow-up. This study supports the feasibility of a support portal and the results indicate the possibility of improving patient outcomes, but further developments are warranted.

USABILITY AND PSYCHOMETRIC PROPERTIES OF THE MOBITEC-GP APP FOR REAL-LIFE MOBILITY ASSESSMENT

Abstract Modern technologies such as the Global Navigation Satellite System and accelerometers which are integrated in every smartphone enable tracking mobility patterns over time, which in turn enables early detection of mobility impairments. Aim of the MOBITEC-GP project was to develop a mobile app which measures walking speed and spatial mobility under everyday conditions. Reliability and validity of the application was tested first (Study 1). The applicability in everyday life and the acceptance by older adults (Study 2) was also investigated. In study 1, participants underwent several supervised walking speed assessments as well as a 1-week life-space assessment during two assessment sessions 9 days apart. Fifty-seven older adults (47.4% male, mean age= 75.3 years) were included in the study. The app showed moderate to excellent test-retest reliability and validity of walking speed measurements of 50 meters and above and of most 1-week life-space parameters, including life-space area, time spent out-of-home, and action range. Study 2 revealed a good usability of the app (mean System Usability Scale score=77.18) among older adults (N=60 with ≥2 chronic conditions) for a 1-week usage of the app including both, a 1-week life space assessment and at least one gait speed measurement in a 30-minute stroll. The MOBITEC-GP app is a reliable, valid and usable tool for the assessment of real-life walking speed (at distances of 50 metres and above) and life-space parameters of older adults. Future studies should look into technical issues more systematically in order to avoid invalid measurements.

Development and Formative Evaluation of a Virtual Exercise Platform for a Community Fitness Center Serving Individuals With Physical Disabilities: Mixed Methods Study.

People with disabilities experience numerous barriers to being physically active, such as transportation issues, a lack of trained exercise professionals who understand disabilities, and facility access. The use of a virtual exercise platform (VEP) may provide an alternative and limit certain barriers. The aim of this mixed method study was to evaluate user interaction (effectiveness, efficiency, and satisfaction), the strengths and weaknesses of the user interface, and the user experience with a VEP. Participants were recruited from a community fitness facility that offers programs for people with disabilities. Inclusion criteria were being older than 18 years, fluent in English, and availability of internet access. Features of the VEP included articles, prerecorded videos, live Zoom classes, web-based class registration, weekly progress tracking, incentives, and surveys. A one-on-one Zoom session was scheduled with each participant, during which they completed certain tasks: (1) create an account or login, (2) register for class, (3) join class, (4) add to calendar, and (5) complete surveys. As participants completed tasks, quantitative observations (time on task, task success, rate of task completion, and number of errors by users, which determined task difficulty), qualitative observations were made and interviews were conducted at the end of the session. The "concurrent think-aloud" method was encouraged by the moderator to gauge participants' thoughts as they worked through testing. Participants also completed the System Usability Scale (SUS) and Questionnaire for User Interface Satisfaction (QUIS). A total of 5 people with disabilities (3 male, 2 female), aged 36-78 (mean 54) years, with education levels from high school to PhD, were recruited. Devices used for testing included a laptop (n=3), a Chromebook (n=1), and a desktop (n=1). All participants completed tasks #1 and #2 without errors but could not complete task #4. One participant completed task #5 with difficulty and another completed task #3 with difficulty. The average time to complete each task was: (1) 82 seconds (55-110), (2) 11 seconds (4-21), (3) 9 seconds (5-27), and (4) 921.5 seconds (840-958). The mean SUS score was 86.5/100, and the mean user QUIS score was 8.08 out of 10. Qualitative observations indicated that the system was simple, user-friendly, and accessible. People with disabilities reported high usability and user satisfaction with the web-based exercise platform, and the system appears to be an efficient and effective physical activity option.

Development of a Digital Health Intervention to Support Patients on a Waitlist for Orthopedic Specialist Care: Co-Design Study.

The demand for orthopedic specialist consultations for patients with osteoarthritis in public hospitals is high and continues to grow. Lengthy waiting times are increasingly affecting patients from low socioeconomic and culturally and linguistically diverse backgrounds who are more likely to rely on public health care. This study aimed to co-design a digital health intervention for patients with OA who are waiting for an orthopedic specialist consultation at a public health service, which is located in local government areas (LGAs) of identified social and economic disadvantage. The stakeholders involved in the co-design process included the research team; end users (patients); clinicians; academic experts; senior hospital staff; and a research, design, and development agency. The iterative co-design process comprised several key phases, including the collation and refinement of evidence-based information by the research team, with assistance from academic experts. Structured interviews with 16 clinicians (female: n=10, 63%; male: n=6, 38%) and 11 end users (age: mean 64.3, SD 7.2 y; female: n=7, 64%; male: n=4, 36%) of 1-hour duration were completed to understand the requirements for the intervention. Weekly workshops were held with key stakeholders throughout development. A different cohort of 15 end users (age: mean 61.5, SD 9.7 y; female: n=12, 80%; male: n=3, 20%) examined the feasibility of the study during a 2-week testing period. The System Usability Scale was used as the primary measure of intervention feasibility. Overall, 7 content modules were developed and refined over several iterations. Key themes highlighted in the clinician and end user interviews were the diverse characteristics of patients, the hierarchical structure with which patients view health practitioners, the importance of delivering information in multiple formats (written, audio, and visual), and access to patient-centered information as early as possible in the health care journey. All content was translated into Vietnamese, the most widely spoken language following English in the local government areas included in this study. Patients with hip and knee osteoarthritis from culturally and linguistically diverse backgrounds tested the feasibility of the intervention. A mean System Usability Scale score of 82.7 (SD 16) was recorded for the intervention, placing its usability in the excellent category. Through the co-design process, we developed an evidence-based, holistic, and patient-centered digital health intervention. The intervention was specifically designed to be used by patients from diverse backgrounds, including those with low health, digital, and written literacy levels. The effectiveness of the intervention in improving the physical and mental health of patients will be determined by a high-quality randomized controlled trial.

Patients' Perception of Usability and Utility of a Digital Health Tool for Electronic Patient-Reported Outcomes Monitoring in Real-Life Hematology Practice: Evidence from the Gimema-Alliance Platform

Background Digital health tools are increasingly being used in oncology practice to better monitor patients' health status. They may include electronic-patient-reported outcome (ePRO) monitoring systems, with automated alerts triggered to the physician depending on specific conditions (e.g., when patients report clinically relevant problems). Although implementation of these tools in real-life practice may offer valuable benefits, it is important to assess their usability and utility from the users' standpoint. Objective The aim of this study was to evaluate the patients' perception of usability and utility of a digital health tool for ePRO monitoring of patients with hematologic malignancies in real-life practice. Methods In December 2020, the GIMEMA Group developed a digital health platform for adult patients with hematologic malignancies (GIMEMA-ALLIANCE platform) with the goal of facilitating patient-centered care. The platform was open to enrollment until December 2022 and involved 26 hospitals. After providing written informed consent, patients received a personal password to access the secure patient portal and complete ePRO questionnaires assessing health-related quality of life (HRQoL) and symptoms (EORTC QLQ-C30 and selected items from the EORTC Item Library). Real-time graphical presentation of PRO results is displayed for both patients and physicians. The platform allows hematologists to receive real-time alerts in the presence of clinically important problems and symptoms. For the purpose of this study, a dedicated section in the patient portal was developed to evaluate the usability and utility of the platform. In this section, patients had the possibility to complete the System Usability Scale (SUS). The SUS is a 10-item widely used questionnaire to evaluate users' perceived system satisfaction. Its score ranges from 0 to 100 and a score ≥70 is considered a threshold for an acceptable usability. Analyses were performed overall and by age group category, based on the median age of the patient population. Patients also completed an ad-hoc survey consisting of 6 items covering aspects on the utility of the platform, for example in favoring shared decision-making or improving the communication with the treating hematologist. Only patients who have completed the ePRO questionnaires at least twice, i.e. those who had the possibility to sufficiently test its functionalities, were considered eligible for completing the survey on usability and utility of the platform. Results Out of the 362 eligible patients, a total of 161 (44%) completed the survey. No difference in age and sex was found between patients who completed or not the survey. The median age of patients who completed the survey was 59 years (IQR: 51 - 67) and 53 (34%) were women. The most prevalent diagnosis was multiple myeloma (n=40, 25%). At the time of survey completion, 69% of the patients were receiving a treatment for their disease. The mean SUS score of the overall population was 80.8 (SD 15.5) and the majority of patients (n=131, 81%) gave a rating ≥70 (the prespecified threshold for the acceptable usability). The mean SUS score for the younger and the older groups was 80.5 (SD 14.9) and 81.4 (SD 16), respectively. Eighty-eight percent of patients agreed or strongly agreed that the platform was easy to use, 83% felt very confident in using the platform, and 72% found the various functionalities offered by the platform well integrated (Figure 1). Positive feedbacks on the utility of the platform were also collected. For example, 71% of patients considered the ePRO questionnaires useful for their health conditions and 63% would recommend its use to other patients. However, amongst the patients who had visits at the clinic (n=127), only 39% reported that their doctor discussed ePRO results with them, and this may explain the lower agreement for some items (Figure 2). For example, 38% of the patients strongly agreed/agreed that the platform helped them to improve the communication with their doctor, while 44% neither agreed or disagreed and 18% strongly disagreed/disagreed. Conclusion This study showed a good usability and utility of the GIMEMA-ALLIANCE platform from the patients' perspective, and this was true for younger and older patients. Future areas of improvement should include actions to facilitate physicians in discussing ePRO results during the clinical encounter with their patients.

Exploring the feasibility of an artificial intelligence based clinical decision support system for cutaneous melanoma detection in primary care – a mixed method study

Objective: Skin examination to detect cutaneous melanomas is commonly performed in primary care. In recent years, clinical decision support systems (CDSS) based on artificial intelligence (AI) have been introduced within several diagnostic fields. Setting: This study employs a variety of qualitative and quantitative methodologies to investigate the feasibility of an AI-based CDSS to detect cutaneous melanoma in primary care. Subjects and Design: Fifteen primary care physicians (PCPs) underwent near-live simulations using the CDSS on a simulated patient, and subsequent individual semi-structured interviews were explored with a hybrid thematic analysis approach. Additionally, twenty-five PCPs performed a reader study (diagnostic assessment on the basis of image interpretation) of 18 dermoscopic images, both with and without help from AI, investigating the value of adding AI support to a PCPs decision. Perceived instrument usability was rated on the System Usability Scale (SUS). Results: From the interviews, the importance of trust in the CDSS emerged as a central concern. Scientific evidence supporting sufficient diagnostic accuracy of the CDSS was expressed as an important factor that could increase trust. Access to AI decision support when evaluating dermoscopic images proved valuable as it formally increased the physician’s diagnostic accuracy. A mean SUS score of 84.8, corresponding to ‘good’ usability, was measured. Conclusion: AI-based CDSS might play an important future role in cutaneous melanoma diagnostics, provided sufficient evidence of diagnostic accuracy and usability supporting its trustworthiness among the users.

Feasibility and Usability of the Job Adjustment Mobile App for Pregnant Women: Longitudinal Observational Study.

Working pregnant women often need to adjust their physically demanding jobs for a healthy pregnancy. However, uncertainty about the extent of these adjustments can hinder their effectiveness. To address this, we developed the Job Adjustment mobile app, which allows users to input job and health details to generate a variety of personalized action plans. As this is the first version of the app, assessing its feasibility and usability is crucial. This study aims to verify the feasibility and usability of the Job Adjustment mobile app. A longitudinal observational study was conducted on pregnant Japanese women who were allowed to use the app anytime from 12 to 34 weeks of gestation; they received reminder emails every 2 weeks encouraging app use. A questionnaire was administered before app use and at 20 and 32 weeks of gestation. Feasibility was evaluated across 4 domains: implementation, demand, acceptability, and adverse events. Implementation was evaluated based on 3 parameters: dropout rate, initial reminder email receipt rate, and adherence rate (measured as pregnant women who used the app at intervals of 2.5 weeks or less). Demand was measured by intervals between use and intervals between log-in, and participants answered 15 questions to assess acceptability. Adverse events were assessed by analyzing the degree of anxiety related to work. Demographic data were analyzed to determine any statistically significant differences in intervals between uses. Usability was evaluated using the System Usability Scale. The analysis included 66 pregnant women, and 61% (n=40) of them were multipara. The dropout rate, adherence rate, and initial reminder email receipt rate were 18% (13/71), 44% (29/66), and 79% (52/66) respectively. The median intervals between use and intervals between log-in were 2.94 (IQR 2.00-5.13) weeks and 2.28 (IQR 1.81-4.00) weeks, respectively. Overall, 60% (35/58) to 90% (52/58) of the participants responded positively to all 15 questions assessing acceptability, and no anxiety regarding work was recorded. The mean System Usability Scale score was 66.1 points. Multipara women had significantly longer intervals between app use compared to primipara women (P=.01). The results demonstrated acceptable levels of feasibility and usability of the app. However, the low adherence rates, especially among multipara women, suggest the need for modifications to reduce the time burden of the app. Further research should explore more effective and acceptable intervals between use and timing, involving a larger sample and accounting for diverse characteristics of pregnant women. UMIN Clinical Trials Registry UMIN000042943; https://tinyurl.com/ydrchfas.

A New Method for Common Femoral Arterial Access Using a Mixed Reality-Assisted Technique on a Phantom Model.

The purpose of this study was to investigate the technical feasibility and usability of a mixed reality (MiR)-assisted common femoral arterial (CFA) access technique using a sonography-assisted registration method. A total of 60 CFA punctures were performed on a phantom model by 2 observers. Thirty punctures were performed using MiR (MiR group) and 30 punctures were performed using a conventional sonography-guided access procedure (control group). In the MiR group, a virtual object was created based on a computed tomography (CT) angiography scan of the model and registered to the physical patient in an MiR environment utilizing a software prototype that allowed registration based on a sonography scan. Positional error assessment encompassed 4 measurements using cone beam CT scans: (1) distance of the needle tip to the centerline, (2) distance of the needle entry site from the mid-level of the ostium of the profound femoral artery, (3) angle of entry of the needle in coronal, and (4) sagittal planes. Technical success rates as well as positional errors were compared between both groups. In addition, the usability of the system was assessed according to the system usability scale (SUS). Technical success was 96.7% and 100% in the MiR and control groups, respectively. The median distance between the needle tip and the centerline was 3.0 (interquartile range [IQR]: 2.0-4.6) in the MiR group and 3.2 mm (IQR: 2.3-3.9) (p=0.63) in the control group. Similarly, the median distance from the needle entry site to the mid-level of the ostium of the profound femoral artery was 3.0 mm (IQR: 2.0-5.0) in the MiR group and 4.5 mm (IQR: 2.0-7.8) (p=0.18) in the control group. The median coronal angles of needle entry were 7.5° (IQR: 6-11) and 6° (IQR: 2-12) (p=0.13), and the median sagittal angles were 50° (IQR: 47-51) and 51° (IQR: 50-55) (p<0.01) in the MiR and control groups, respectively. The mean SUS score provided by both observers was 51.3. The feasibility of an MiR-assisted CFA access technique could be demonstrated on a phantom model. Further studies are needed to investigate the technique beyond phantom model experiments and in different anatomical settings. This study demonstrates the technical feasibility of a Mixed-Reality-assisted common femoral arterial access procedure on a phantom model. The positional accuracy was comparable to a conventional sonography-guided technique. However, there are several limitations that need to be resolved prior to potential implementation into clinical practice. Further studies are needed to investigate its performance beyond phantom model experiments and the prototypical application requires further technical refinement to increase its usability.

Virtual Reality Simulation in Nursing and Midwifery Education: A Usability Study.

Virtual reality simulation offers students the opportunity to acquire clinical and psychomotor skills in a safe and interactive environment. This study describes the usability of virtual reality simulation among undergraduate nursing and midwifery students. Participants were recruited using convenience and snowball sampling and engaged in a 20-minute virtual reality simulation scenario of their choice. They then completed a 21-item survey comprising a sociodemographic questionnaire, the System Usability Scale, a satisfaction questionnaire, and open-ended questions. Quantitative data were analyzed using descriptive statistics, and qualitative data were analyzed using deductive content analysis. Forty-three students participated in this study. The mean (SD) System Usability Scale score was 75.87 (13.7), indicating that virtual reality simulation was acceptable. Almost all participants were either "extremely satisfied" or "somewhat satisfied" with virtual reality simulation, which was perceived as informative and enjoyable, fostering safe and self-directed learning without causing patient harm. Participants recommended using virtual reality simulation to practice clinical skills, prepare for clinical placements, and learn about rare clinical situations. Virtual reality simulation needs to be underpinned by a strong pedagogy and aligned with learning outcomes. Educators and students should be trained in virtual reality simulation prior to its integration into the curriculum.

Augmented Reality-Guided Apicoectomy Based on Maxillofacial CBCT Scans.

Implementation of augmented reality (AR) image guidance systems using preoperative cone beam computed tomography (CBCT) scans in apicoectomies promises to help surgeons overcome iatrogenic complications associated with this procedure. This study aims to evaluate the intraoperative feasibility and usability of HoloLens 2, an established AR image guidance device, in the context of apicoectomies. Three experienced surgeons carried out four AR-guided apicoectomies each on human cadaver head specimens. Preparation and operating times of each procedure, as well as the subjective usability of HoloLens for AR image guidance in apicoectomies using the System Usability Scale (SUS), were measured. In total, twelve AR-guided apicoectomies on six human cadaver head specimens were performed (n = 12). The average preparation time amounted to 162 (±34) s. The surgical procedure itself took on average 9 (±2) min. There was no statistically significant difference between the three surgeons. Quantification of the usability of HoloLens revealed a mean SUS score of 80.4 (±6.8), indicating an "excellent" usability level. In conclusion, this study implies the suitability, practicality, and simplicity of AR image guidance systems such as the HoloLens in apicoectomies and advocates their routine implementation.

Diagnostic Performance, Triage Safety, and Usability of a Clinical Decision Support System Within a University Hospital Emergency Department: Algorithm Performance and Usability Study.

Computerized clinical decision support systems (CDSSs) are increasingly adopted in health care to optimize resources and streamline patient flow. However, they often lack scientific validation against standard medical care. The purpose of this study was to assess the performance, safety, and usability of a CDSS in a university hospital emergency department setting in Kuopio, Finland. Patients entering the emergency department were asked to voluntarily participate in this study. Patients aged 17 years or younger, patients with cognitive impairments, and patients who entered the unit in an ambulance or with the need for immediate care were excluded. Patients completed the CDSS web-based form and usability questionnaire when waiting for the triage nurse's evaluation. The CDSS data were anonymized and did not affect the patients' usual evaluation or treatment. Retrospectively, 2 medical doctors evaluated the urgency of each patient's condition by using the triage nurse's information, and urgent and nonurgent groups were created. The International Statistical Classification of Diseases, Tenth Revision diagnoses were collected from the electronic health records. Usability was assessed by using a positive version of the System Usability Scale questionnaire. In total, our analyses included 248 patients. Regarding urgency, the mean sensitivities were 85% and 19%, respectively, for urgent and nonurgent cases when assessing the performance of CDSS evaluations in comparison to that of physicians. The mean sensitivities were 85% and 35%, respectively, when comparing the evaluations between the two physicians. Our CDSS did not miss any cases that were evaluated to be emergencies by physicians; thus, all emergency cases evaluated by physicians were evaluated as either urgent cases or emergency cases by the CDSS. In differential diagnosis, the CDSS had an exact match accuracy of 45.5% (97/213). The usability was good, with a mean System Usability Scale score of 78.2 (SD 16.8). In a university hospital emergency department setting with a large real-world population, our CDSS was found to be equally as sensitive in urgent patient cases as physicians and was found to have an acceptable differential diagnosis accuracy, with good usability. These results suggest that this CDSS can be safely assessed further in a real-world setting. A CDSS could accelerate triage by providing patient-provided data in advance of patients' initial consultations and categorize patient cases as urgent and nonurgent cases upon patients' arrival to the emergency department.